Skip to content

Add retries for token creation in tests#793

Open
vladem wants to merge 2 commits into
awslabs:mainfrom
vladem:retry-create-token-in-tests
Open

Add retries for token creation in tests#793
vladem wants to merge 2 commits into
awslabs:mainfrom
vladem:retry-create-token-in-tests

Conversation

@vladem
Copy link
Copy Markdown
Collaborator

@vladem vladem commented May 14, 2026

Issue #, if available:

Description of changes:

Retries CreateToken in tests for up to ~15s to handle transient API server errors (e.g. gRPC connection closing during token signing).

Example failure: https://github.com/awslabs/mountpoint-s3-csi-driver/actions/runs/25864837638/job/76005435210?pr=790#step:9:3160

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Add retries for token creation in tests
ee2d7d4 
Signed-off-by: Vlad Volodkin <vlaad@amazon.com>
@vladem vladem force-pushed the retry-create-token-in-tests branch from 870f355 to ee2d7d4 Compare May 14, 2026 17:10
@vladem vladem temporarily deployed to approval-gate May 14, 2026 17:10 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to rosa-untrusted May 14, 2026 17:11 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:11 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to rosa-untrusted May 14, 2026 17:19 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 14, 2026 18:11 — with GitHub Actions Inactive
@vladem vladem marked this pull request as ready for review May 15, 2026 17:01
@vladem vladem requested a review from a team as a code owner May 15, 2026 17:01
@vladem vladem temporarily deployed to approval-gate May 15, 2026 17:02 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 15, 2026 17:03 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to rosa-untrusted May 15, 2026 17:03 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 15, 2026 17:10 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 15, 2026 17:10 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 15, 2026 17:10 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 15, 2026 17:10 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 15, 2026 17:10 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 15, 2026 17:10 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 15, 2026 17:10 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 15, 2026 17:10 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 15, 2026 17:10 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 15, 2026 17:10 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 15, 2026 17:10 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to us-east-1-untrusted May 15, 2026 17:10 — with GitHub Actions Inactive
jet-tong
jet-tong approved these changes May 18, 2026
View reviewed changes
Comment thread tests/e2e-kubernetes/testsuites/credentials.go
})
}

// Retries CreateToken for up to ~15s to handle transient API server errors
Copy link
Copy Markdown
Collaborator

@jet-tong jet-tong May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

np: found out Steps is attempt count not retry count, so 5 steps = 4 sleeps = 0.5+1+2+4 seconds. If Duration is 1s then 1+2+4+8=15 :)

Comment thread tests/e2e-kubernetes/testsuites/credentials.go
backoff := wait.Backoff{Steps: 5, Duration: 500 * time.Millisecond, Factor: 2.0, Jitter: 0.1}
err := k8sretry.OnError(backoff, apierrors.IsInternalError, func() error {
var err error
serviceAccountToken, err = saClient.CreateToken(ctx, sa.Name, tokenRequest, metav1.CreateOptions{})
Copy link
Copy Markdown
Collaborator

@jet-tong jet-tong May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: maybe we can add framework.Logf line here so we can see how much it helps later?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jet-tong jet-tong jet-tong approved these changes

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vladem @jet-tong